Skip to content

Fix bignum zeroization, MAXQ10XX shared mutex setup and template HardwareModuleName decoding - #12067

Open
LinuxJedi wants to merge 7 commits into
wolfSSL:masterfrom
LinuxJedi:fenrir-wolfcrypt-batch
Open

LinuxJedi wants to merge 7 commits into
wolfSSL:masterfrom
LinuxJedi:fenrir-wolfcrypt-batch

Conversation

@LinuxJedi

@LinuxJedi LinuxJedi commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes for seven wolfCrypt issues, one commit each. Based on master, not stacked on another PR.

Bignum zeroization

  • sp_clear() (sp-math-all). Outside FIPS builds it zeroed only the digits below used, with plain stores. Digits above used can still hold part of an earlier, larger value, and the stores can be removed as dead. It now wipes all size digits with ForceZero(), as sp_forcezero() and FIPS builds already did. This relies on size matching the storage. The template CRL Number decode initialized a key-sized sp_int a second time through GetInt(), which calls mp_init() and so sets size to the maximum. With HAVE_WOLF_BIGINT, the mp_clear() done when the bigint allocation fails then overran the CRL Number's storage. It now decodes into the initialized mp_int with GetASN_MP_Inited, as DecodeECC_DSA_Sig_Ex() already does for r and s.

    Every other INIT_MP_INT_SIZE() and mp_init_size() site, and the functions those mp_ints are passed to, was checked for the same pattern. The one other case, wc_ecc_verify_hash() in NO_ASN builds, has since been fixed on master.

  • mp_clear() (heap math). Freed the digit buffer after wiping only the digits below used, also with plain stores. mp_forcezero() now wipes all alloc digits with ForceZero(), and mp_clear() calls it in all builds, as FIPS builds already did.

  • mp_grow() (heap math). Grew the digit array with XREALLOC(), so each time realloc() moved the block, the old copy of the value was freed without being wiped. With MP_PREC set to 1 this happens often, including for private key values. It now allocates the new array, copies the digits across, and wipes and frees the old one.

  • fp_clear() (fastmath). Used a plain memset(), which the compiler can drop because fp_clear() is usually the last use of the fp_int. With clang -O2, after the last Miller-Rabin round all three wipes in mp_prime_is_prime() and four of the five in mp_prime_is_prime_ex() are removed, leaving values derived from RSA candidate primes on the stack. It now calls fp_forcezero() in all builds.

MAXQ10XX shared hardware mutex

  • Init error paths. wc_InitMutex() leaked the shared memory descriptor and mapping on failure, and set wcCryptHwSharedMutexPtr before the mutex was initialized. A failure after creating the object left it in place without an initialized mutex, and the next init then used it as is. maxq_LockMutex() also leaked the per-thread lock count when pthread_setspecific() failed.
  • Create/init race between processes. A process that opened the object after another one created it, but before that one initialized the mutex, skipped initialization. The creator's pthread_mutex_init() could then reset a lock the other process held, leaving both holding the mutex. Two processes that both found no object also raced on O_EXCL, and the loser failed. The object is now opened with O_CREAT, and a record lock on it (fcntl(F_SETLKW)) is held while it is checked and initialized. The system releases that lock if its holder exits, so a process killed during initialization does not block the others. A process-local mutex covers the threads of a process, which record locks do not exclude. The object's size records whether the mutex is ready: one byte larger than the mutex during initialization, the size of the mutex once ready. An empty object, or one left part way through initialization, is initialized again. A ready object has the same layout and size as before, so one left in /dev/shm by an earlier build is still used.
  • Holder exiting with the mutex locked. A process that exited while holding the shared mutex, part way through using the MAXQ10xx, blocked every other process for good. The mutex is now robust: the next process to lock it gets EOWNERDEAD, marks it consistent and carries on. A mutex initialized by an earlier build is not robust, and works as before until its object is removed. This is in the same commit as the race fix, which also gives maxq_CryptHwMutexTryLock() a prototype, as clang warns without one.

ASN.1 template parser

  • HardwareModuleName otherName. The template DecodeSEP() did not reject a second HardwareModuleName, so each one overwrote and leaked the previous hwType and hwSerialNum. It also never decoded the HardwareModuleName contents: both fields came out zero-length, and the parse failed with MEMORY_E wherever a zero-byte allocation returns NULL. The contents are now decoded with a template of their own, and a second HardwareModuleName is rejected with ASN_PARSE_E, as the original parser already does. Other otherName types with a SEQUENCE value, such as permanentIdentifier, are still accepted.

Cost of the wider wipes, in callgrind instruction counts on arm64 Linux (gcc -O2, ECDH + ECDSA sign/verify on P-256, RSA-2048 and FFDHE-2048): sp-math-all ECC +1.5%, fastmath and heap math ECC about +0.5%, RSA and DH within 0.3%. The mp_grow() change adds nothing: heap math runs 0.6% to 2% fewer instructions than with XREALLOC().

Testing

  • New API tests, each of which fails without its fix: test_DecodeAltNames_dup_hw_module_name, test_mp_clear_zeroize (sp-math-all, fastmath and heap math) and test_mp_grow_zeroize (heap math).
  • make check on the default build and --enable-all. unit.test on --enable-heapmath and --enable-fastmath.
  • ASan: testwolfcrypt on the default build, --enable-smallstack, --enable-all and --enable-all --enable-smallstack, plus unit.test on both --enable-all builds. testwolfcrypt and unit.test on --enable-heapmath, with and without --enable-smallstack. The CRL Number overflow was reproduced with ASan on --enable-crl --enable-pkcs11 (which defines HAVE_WOLF_BIGINT) by failing each allocation in turn in wolfSSL_CertManagerLoadCRLBuffer(). Before the change, failing the bigint allocation overflows the stack in sp_clear(). After it, no failure point reports an error.
  • The HardwareModuleName test also passes with --enable-sep --enable-asn=original, with no leaks reported by WOLFSSL_TRACK_MEMORY.
  • MAXQ10XX: the mutex code builds without the vendor SDK (--enable-cryptonly with -DMAXQ10XX_MUTEX -DWOLFSSL_CRYPT_HW_MUTEX=1). It was tested on arm64 Linux with a harness that injects failures into the pthread calls with -Wl,--wrap and runs several processes. Before these changes, it reproduces the leaked lock count, the leftover object being used without initialization, the create/init race (both processes holding the mutex), and an empty or undersized object being used as is. It also checks that a process killed while initializing the mutex does not block the next process, which initializes it again, and that a process killed while holding the mutex does not block the next one, whether that one locks or trylocks it. Before these changes, the next process hangs in both cases. If the mutex cannot be made consistent, locking it fails rather than blocks. All pass with this branch. 16 processes making 2000 locked increments each of a shared counter reach the exact total, and when 8 of them are killed part way, the other 8 still finish. The mutex code builds with -Wall -Wextra -Werror under gcc and clang.

Checklist

  • added tests
  • updated/added doxygen
  • updated appropriate READMEs
  • Updated manual and documentation

Copilot AI balanced review requested due to automatic review settings November 9, 2026 17:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@LinuxJedi

Copy link
Copy Markdown
Member Author

@claude review

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m0plus

  • FLASH: .text -24 B (-0.0%, 71,171 B / 262,144 B, total: 27% used)

gcc-arm-cortex-m3

  • FLASH: .text -28 B (-0.0%, 132,201 B / 262,144 B, total: 50% used)

gcc-arm-cortex-m4-crypto-only

  • FLASH: .text -64 B (-0.0%, 182,227 B / 262,144 B, total: 70% used)

gcc-arm-cortex-m4-min-ecc

  • FLASH: .text -64 B (-0.1%, 68,377 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-openssl-compat

  • FLASH: .text +128 B (+0.0%, 802,244 B / 1,048,576 B, total: 77% used)

gcc-arm-cortex-m4-sp-math

  • FLASH: .text -64 B (-0.1%, 68,377 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-tls12

  • FLASH: .text -64 B (-0.0%, 132,945 B / 262,144 B, total: 51% used)

stm32-sim-stm32h753

  • FLASH: .text +64 B (+0.0%, 204,532 B / 2,097,152 B, total: 10% used)

zephyr-frdm_rw612

  • FLASH: datas +16 B, log_const_area -8 B, rodata +36 B, text +10,520 B (+3.7%, 293,028 B / 67,108,864 B, total: 0% used)
  • RAM: bss +49 B, datas +16 B (+0.1%, 58,423 B / 983,040 B, total: 6% used)

No memory changes detected for:


Catch memory regressions in CI with MemBrowse. Free for open source.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread wolfcrypt/src/sp_int.c Resolved
Comment thread wolfcrypt/src/wc_port.c Resolved
@LinuxJedi
LinuxJedi force-pushed the fenrir-wolfcrypt-batch branch from 1e26eb7 to 04a6f4a Compare November 9, 2026 17:36

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #12067

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-bugs
Not reviewed (no scan target covers them): tests/api/test_asn.h, tests/api/test_wolfmath.h
Coverage: 8 of 8 in-scope changed file(s) opened by the reviewer

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Review tier: Max

Comment thread wolfcrypt/src/wc_port.c Resolved
@LinuxJedi
LinuxJedi force-pushed the fenrir-wolfcrypt-batch branch 3 times, most recently from b3bbf42 to ea5586d Compare November 10, 2026 11:48

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #12067

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-bugs
Not reviewed (no scan target covers them): tests/api/test_asn.h, tests/api/test_wolfmath.h
Coverage: 7 of 7 in-scope changed file(s) opened by the reviewer

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

@wolfSSL-Fenrir-bot
wolfSSL-Fenrir-bot dismissed their stale review November 10, 2026 13:13

Fenrir's latest completed scan found no issues; clearing the prior automated change request.

@LinuxJedi LinuxJedi assigned wolfSSL-Bot and unassigned LinuxJedi Nov 10, 2026
The template DecodeSEP() assigned cert->hwType and cert->hwSerialNum
without checking whether they were already set. A subjectAltName with
several HardwareModuleName otherNames overwrote the buffers of the
previous one on each entry, and FreeDecodedCert() freed only the last
pair, leaking memory on every parse of such a certificate. Reject a
second HardwareModuleName with ASN_PARSE_E, as the original parser
already does.

Writing a test for this showed that the template never decoded the
HardwareModuleName contents. The OtherName template takes the value's
SEQUENCE as a whole, so its hwType and hwSerialNum child items were
never filled in. Both were stored as zero-length buffers, and the parse
failed with MEMORY_E wherever a zero-byte allocation returns NULL.
Decode the SEQUENCE contents in DecodeSEP() with a template of their
own, once the type-id is known to be a HardwareModuleName, and set
oidIgnoreType on hwType before parsing. Other otherName types that have
a SEQUENCE value are still accepted as before.

Add test_DecodeAltNames_dup_hw_module_name. It checks that a single
HardwareModuleName is decoded, that a second one is rejected, and that
a permanentIdentifier otherName is still accepted.
wc_InitMutex() for the MAXQ10XX shared hardware mutex returned on
failure without closing the shared memory descriptor or unmapping the
mapping, and it set wcCryptHwSharedMutexPtr before the mutex was
initialized. When it created the shared memory object and then failed,
the object was left in place without an initialized mutex. The next
init attempt opened the existing object, skipped initialization
because it had not created it, and returned success, so callers locked
a mutex that was never initialized.

Close the descriptor on every path, unmap on failure, and destroy the
mutex attributes. Remove a newly created object whose mutex was not
initialized. Publish wcCryptHwSharedMutexPtr only once initialization
has succeeded.

Also free the per-thread lock count in maxq_LockMutex() when
pthread_setspecific() fails, instead of leaking it.
Outside FIPS builds, sp_clear() zeroed only the digits below used, with
plain stores. Data is not zeroed when used decreases, so digits above
used could still hold part of an earlier, larger value, and the stores
could be removed as dead when the sp_int was not used again. Callers
use mp_clear() on secret values, so these could be left in memory.

Make sp_clear() wipe all a->size digits with ForceZero(), as
sp_forcezero() and FIPS builds already did, and have sp_forcezero()
call it.

sp_clear() now relies on size matching the storage. The template CRL
Number decode used GetInt(), which initializes the mp_int it decodes
into a second time, setting size to the maximum. With HAVE_WOLF_BIGINT,
the mp_clear() done when the bigint allocation fails then overran the
CRL Number's storage. Decode into the initialized mp_int instead.

Add test_mp_clear_zeroize, which checks that mp_clear() leaves no
non-zero digits after the value has been made smaller.
Outside FIPS builds, the heap math mp_clear() zeroed only the digits
below used, with plain stores, and then freed the digit buffer. Any
digits from used up to alloc were freed without being wiped, and the
stores could be removed as dead because the buffer is freed straight
after. mp_forcezero() also wiped only the used digits.

Make mp_forcezero() wipe all alloc digits with ForceZero(), and have
mp_clear() call it in all builds, as FIPS builds already did. Keep the
HAVE_WOLF_BIGINT handling from mp_clear(), which frees the raw buffer
even when dp is NULL.

Extend test_mp_clear_zeroize to heap math. It uses allocator callbacks
to check that the freed digit buffer has been wiped in full.
Outside FIPS builds, fp_clear() wiped the digits with a plain memset.
fp_clear() is normally the last use of the fp_int, so the compiler may
remove the memset as a dead store. For example, with clang -O2 the
Miller-Rabin code is inlined into mp_prime_is_prime() and
mp_prime_is_prime_ex(). After the last round, all three fp_clear()
wipes in mp_prime_is_prime() and four of the five in
mp_prime_is_prime_ex() are removed. RSA key generation uses these to
test candidate primes, so values derived from the secret primes, such
as p - 1, were left on the stack.

Make fp_clear() call fp_forcezero() in all builds, as FIPS builds
already did. It wipes the same digits with ForceZero(), whose barrier
keeps the stores.
wc_InitMutex() for the MAXQ10xx hardware mutex created the shared memory
object and initialized the process-shared mutex in it as separate steps,
and decided whether to initialize it from which of two shm_open() calls
succeeded. A process that opened the object after another one created it,
but before that one initialized the mutex, skipped initialization and
could take a lock that the creator's pthread_mutex_init() then reset,
leaving both processes holding the mutex. Two processes that both found
no object also raced on O_EXCL, and the loser failed outright.

Open the object with O_CREAT and hold a record lock on it while checking
and initializing it, with a process-local mutex for the threads of a
process, which record locks do not exclude. The system releases the
record lock if its holder exits, so a process killed during
initialization does not block the others.

The size of the object now records whether the mutex is ready: it is one
byte larger than the mutex while being initialized, and the size of the
mutex once ready. An object that is empty, or that a process left part
way through initializing, is initialized again, so a failed
initialization no longer needs to remove the object. An existing object
is no longer truncated, and one of any other size is rejected. A ready
object has the same layout and size as before, so one left in place by
an earlier build is still used.

Make the mutex robust, so that a process that exits while holding it,
part way through using the MAXQ10xx, no longer blocks every other
process: the next process to lock it gets EOWNERDEAD, marks it
consistent and carries on. A mutex initialized by an earlier build is
not robust, and works as before until its object is removed.

Also give maxq_CryptHwMutexTryLock() a prototype, which clang warns
about without.
mp_grow() enlarged the digit array with XREALLOC(). When realloc() cannot
grow a block in place, it copies it to a new block and frees the original
without wiping it, leaving a copy of the value in freed memory. mp_grow()
grows mp_ints that hold private key values and intermediates, and with
MP_PREC set to 1 it reallocates often.

Allocate the new array, copy the digits across, then wipe and free the
old one.

Add a test, with a realloc callback that always moves the block, that
checks mp_grow() wipes the array it replaces and keeps the value.
@LinuxJedi
LinuxJedi force-pushed the fenrir-wolfcrypt-batch branch from ea5586d to a86f6b8 Compare November 11, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants